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Pennsylvania 
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1. Introduction 

In an evolving public education landscape, charter school education reaches an increasing number of 
students each year. While the expansion of charter schools is evident, questions about their efficacy 
persist. Active debate has occurred in Pennsylvania continuously since the passage of the law 
authorizing charter schools in 1997. Charter school advocates hail the benefits of the sector such as 
increasing parental choices and introducing new school models. Opponents decry the reallocation of 
funds away from district schools as an existential threat to district organizations and the mismatch 
between district and charter student profiles as evidence of charters’ neglecting hard-to-serve 
students. Only a fraction of that debate is grounded in well-researched evidence about charter schools, 
their practices, and their impact on student outcomes. 

The need for evidence about charter school performance is especially strong in Pennsylvania. The 
charter school law in Pennsylvania has remained largely unchanged since its passage in 1997. There 
were minor amendments to the law, including the amendment to authorize cyber charter schools in 
2002.^ Since the amendment authorizing cyber charter schools, there have been many efforts to enforce 
existing regulations of charter schools to hold all charters accountable. Efforts to strengthen the 
regulatory environment for charter schools have been rebuffed, but the debate around accountability 
of charter schools in Pennsylvania continues. 

According to the Center for Rural Pennsylvania, charter school enrollment has grown dramatically since 
the mid-2000s, with noteworthy expansion in both urban and rural areas. In addition, Pennsylvania 
experienced a 75 percent increase in online charter school enrollment between 2006-2007 and 2010- 
2011.^ Currently one quarter of Pennsylvania’s charter school students enroll in online charter schools. 
These trends motivate the current study. 

This report provides evidence for the effect of charter schools on students’ performance in 
Pennsylvania over fours year of schooling, beginning with the 2013-2014 school year and ending with 


^ Charter School Law, Pennsylvania General Assembly, 1949 Act 14 (1997). 

^ Schafft, K., Frankenberg, E., Fuller, Ed., Flartman, W., Kotok, S., Mann, B., Penn State University, Department of 
Education Policy Studies. Assessing the Enrollment Trends and Financial Impact of Charter Schools on Rural and 
Non-Rural School Districts in Pennsylvania (2014). 

http://www.rural.palegislature.us/documents/reports/Charter_School_2014.pdf. 
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the 2016-2017 school year. This is an update to an earlier study released in 2011.^ The 2011 study found 
across all charter schools, student academic progress in reading and math lagged behind identical 
peers in traditional public schools (TPS). The 2011 findings also showed wide variation in student and 
school performance, with a quarter of charter schools outperforming their local school options in 
reading and over half outpacing their local TPS in math. Of particular note, the 2011 study was the first 
time the differences in student academic progress for online charter schools and brick-and-mortar 
charter schools were compared; online charter schools posted significantly smaller learning gains than 
were seen in other charter schools.'^ This study updates the earlier analyses with contemporary data. 

With cooperation from the Pennsylvania Department of Education (PDOE), CREDO obtained historical 
sets of student-level administrative records. The support of PDOE staff was critical to CREDO’S 
understanding of the character and quality of the data we received. However, it is important to note 
that those interactions with the department dealt only with technical issues related to the data. CREDO 
has developed the findings and conclusions presented here independently. 

In this report, we present the results from three sets of analysis. We first present findings regarding the 
effects of charter schools on student academic performance for the period 2013-2014 to 2016-2017. 
These results are expressed in terms of the academic progress that a typical charter school student in 
Pennsylvania would realize from a year of enrollment in a charter school. To help the non-technical 
reader grasp the findings, we translate the scientific estimates into estimated days of learning based 
on the foundation of a 180-day school year. 

The second set of analysis looks at the performance of students by school and presents school average 
results. These findings are important to understand the range of performance at the school level. 

The third set of analysis illustrates the impact of online charter schools in Pennsylvania, also referred 
to as cyber charter schools. Students attending online charter schools represent a quarter of all 
students attending charter schools in Pennsylvania. Online charter schools serve students with 
different characteristics and deliver curriculum differently than brick-and-mortar charter schools. Our 
analysis focuses on charterschoolsthat provide full-time online education and excludes programs that 
incorporate online instruction as a portion of a blended educational model. 

The analysis shows that in a year's time, the typical charter school student in Pennsylvania makes 
similar progress in reading and weaker growth in math compared to the educational gains that the 
students would have had in a traditional public school (TPS). Thinking of a 180-day school year as "one 
year of learning", an average Pennsylvania charter student experiences weaker annual growth in math 
equivalent to 30 fewer days of learning. Our online charter school analysis reveals that attending an 


^ Charter School Performance in Pennsylvania, CREDO (Center for Research on Education Outcomes), Stanford 
University, April 6, 2011, http://credo.stanford.edu/reports/PA%20State%20Report_20110404_FINAL.pdf 
Student academic progress is the change in a student’s academic achievement from one year to the next. We 
also refer to this change in knowledge as “gains” or “growth”, not to be confused with the Pennsylvania value- 
added model of student performance. 


credo.stanford.edu 


2 



online charter school leads to substantially negative learning gains in both reading and math, which 
negatively affect the overall charter impact on student progress. 

2. Study Approach 

This study of charter schools in Pennsylvania focuses on the academic progress (growth) of enrolled 
and tested students in Pennsylvania’s charter schools. Whatever else charter schools may provide their 
students, their contributions to students’ readiness for secondary education, high school graduation, 
and post-secondary life remain of paramount importance. Furthermore, current data limitations 
prevent the inclusion of non-academic outcomes in this analysis. 

To study academic performance of charter students in Pennsylvania, we relied on scores students 
received on Pennsylvania state standardized achievement tests. Achievement tests capture what a 
student knows at a point in time. These test results were fitted into a bell curve format enabling us to 
see how students moved from year to year in terms of academic performance. Two successive test 
scores allow us to see how much progress a student makes over a one-year period; this is also known 
as a growth score or learning gain. Growth scores allow us to zero in on the contributions of schools 
separately from other things that affect point-in-time scores. The parsed effect of schools in turn gives 
us the chance to see how students’ academic progress changes as the conditions of their education 
transform. This is the analytic foundation for our examination of the academic impact of enrollment in 
charter schools. 

We employ the Virtual Control Record (VCR) method developed by CREDO in our analysis.® We strive to 
build a VCR for each charter school student. A VCR, or a “virtual twin”, is a synthesis of the actual 
academic experiences of up to seven students who share identical characteristics to the charter school 
student, except for the fact that the VCR students attend a TPS that each charter school’s students 
would have attended if not enrolled in the charter school. This synthesized record is then used as the 
counterfactual condition to the charter school student’s performance. 


® Davis, D. H., & Raymond, M. E. (2012). Choices for studying choice: Assessing charter school effectiveness using 
two quasi-experimental methods. Economics of Education Review, 31{2), 225-236. 


credo.stanford.edu 


3 



Our approach is displayed in Figure 1. We identify all the traditional public schools whose students 
transfer to a given charter school; each of these schools is designated as a “feeder school.” Using the 
records of the students in those schools in the year prior to the test year of interest (to), CREDO selects 
all of the available TPS students who match each charter school student. 


Match factors include: 


• Grade level 

• Gender 

• Race/Ethnicity 

• Poverty Status 

• English Language Learner Status 

• Special Education Status 

• Prior test score on Pennsylvania state achievement tests 
Figure 1: CREDO Virtual Control Record Methodology 
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At the point of selection as a VCR-eligible TPS student, all candidates and the individual charter school 
student have identical traits and matching baseline test scores. The focus then moves to the 
subsequent year, ti. The scores from this test year of interest (ti) for as many as seven VCR-eligible TPS 
students are then averaged and a Virtual Control Record is produced. The VCR produces a score for the 
test year of interest that corresponds to the expected result a charter student would have realized had 
he or she attended one of the traditional public schools. 


credo.stanford.edu 


4 









The above VCR method has been used in previous CREDO publications. We make two changes to the 
approach in this study. First, in our previous reports, if a charter student can be tracked for multiple 
periods in the study window, we matched the student for all the periods using the records in the year 
prior to the first growth period. In this study, we match the student period by period to conform to the 
new baseline equivalence criteria specified in Procedures Handbook Version 4.0 of What Works 
Clearinghouse (WWC).® Altering the match in this way means that caution is advised when comparing 
findings in this study and previous reports. Second, the United States Department of Agriculture phased 
in the Community Eligibility Provision (CEP) in Pennsylvania and other states during the study period. 
The CEP allows schools and local education agencies with a minimum Identified Student Percentage 
(40 percent or higher) to provide free breakfast and lunch to all students. To minimize over¬ 
identification of students living in poverty in the analysis, we drop from the list of feeder schools a very 
small number of TPS if their share of the students identified as economically disadvantaged by the state 
was 100 percent and represented a jump by 35 percentage points or more from the previous year. As 
Appendix Table 2 shows, restricting the feeder list did not affect the percentage of charter students for 
whom a VCR match was possible. It was possible to create virtual matches for 84 percent of 
observations of tested charter school students in both reading and math. 

Using statistical methods, we isolate the contributions of schools from other social or programmatic 
influences on a student's growth. Student growth data are analyzed in standard deviation units so that 
the results can be assessed for statistical differences. All the findings that follow are reported as the 
average one-year growth of charter school students relative to their VCR-based comparisons. With 
four years of student records in this study, it is possible to create three periods of academic growth. 

To assist the reader in interpreting the meaning of growth, we include an estimate of the number of 
days of learning required to achieve growth of particular units of standard deviations. This estimate 
was calculated by Dr. Eric Hanushek and Dr. Margaret Raymond based on the 2017 National Assessment 
of Educational Progress (NAEP) test scores.^ Using a standard 180-day school year, each one standard 
deviation (s.d.) change in effect size is equivalent to 590 days of learning. 


® What Works Clearinghouse (2017). Procedures Handbook Version 4.0. 
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_handbook_v4.pdf. 
^ Detailed information about the 2017 NAEP test scores can be accessed via 
https://www.nationsreportcard.gov/reading_2017/?grade=4 and 
https://www.nationsreportcard.gov/math_2017/?grade=4. 
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3. Pennsylvania Charter School Demographics 

The total number of charter schools in the state of Pennsylvania has remained stable across the study 
period. Figure 2 notes the newly opened, continuing, and closed charter school campuses from the 
2013-14 school year to the 2016-17 year according to the National Center for Education Statistics 
(NCES). ® The figure shows the consistency of total charter schools in Pennsylvania over four years of 
time. 

Figure 2: Opened, Continuing, and Closed Charter Campuses, 2013-14 to 2016-17 
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The small number of new charter openings or charter school closures creates a stable charter school 
sector over the study period. Our analysis begins with a total of 181 charter schools in the 2013-14 
school year. The 2014-15 school year saw 13 schools closed and four schools opened. In 2015-16, there 
were four new openings and four closures. In the 2016-17 school year, nine new schools opened, while 
only two schools closed, leaving the total amount of charter schools to 181. 

As a general matter, the demographics of the charter schools may not mirror those of the TPS of 
Pennsylvania as a whole. This is because charter schools are able to choose their location and thus may 


® The data were retrieved from “Public Elementary/Secondary School Universe Survey Data,” National Center for 
Education Statistics, https://nces.ed.gov/ccd/pubschuniv.asp. “Opened schools” indicates schools opened as 
new schools in the fall of the displayed year. “Continuing schools” indicates schools that were opened prior to 
the fall of the displayed year and remain open into the next school year (i.e. a school listed as continuing in the 
2014-15 column opened some time prior to 2014-15 and did not close in 2014-15) “Closed schools” indicates 
schools that ceased operation by the spring of the displayed year (i.e. a school listed as closed in the 2014-15 
column had its last year of operation in 2014-15 and closed at the end of that school year). 
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attract a set of students who differ demographically from the overall community profile. Furthermore, 
charter schools may offer different academic programs and alternate school models which may 
disproportionately attract particular groups of students relative to TPS. In addition, parents and 
students choose to attend charter schools for a variety of reasons, such as location, school safety, small 
school size, academic focus, or special interest programs. The cumulative result of all these forces is 
that the student populations at charter schools and their TPS feeders may differ. 

Table 1 compares student populations in all Pennsylvania traditional public schools (TPS), in those TPS 
that comprise the set of charter feeder schools, and in the charter schools themselves in the 2015-16 
school year. 
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Table 1: Demographic Comparison of Students in TPS, Feeders, and Charters: 2015-16 



TPS 

Feeders 

Charters 


Number of schools 

2,844 

1,548 

175 

Average enrollment per school 

553 

572 

748 

Total number of students enrolled 

1,573,535 

886,205 

130,940 

Students in Poverty 

43% 

50% 

66% 

English Language Learners 

3% 

4% 

3% 

Special Education Students 

15% 

16% 

16% 

White Students 

70% 

64% 

35% 

Black Students 

12% 

16% 

43% 

Hispanic Students 

10% 

12% 

16% 

Asian/Pacific Islander Students 

4% 

4% 

3% 

Native American Students 

0% 

0% 

0% 

Multi-Racial Students 

3% 

4% 

4% 


The data in Table 1 show the demographic profile of feeder schools is somewhat similar to that of TPS 
as a whole. The percentage of students in poverty. Black students, and Hispanic students is higher in 
feeders than in TPS, while the percentage of White students in feeders is lower than in TPS. The 
demographics in charter schools, however, are very different from that of TPS. Charter schools have a 
higher percentage of students in poverty as well as a higher percentage of Black students. The 
percentage of White students in charter schools is lower in charter schools than in TPS. ® 

Policymakers and stakeholders continue to examine the degree to which students with special needs 
enroll in charter schools. The proportion of students in charter schools receiving special education 
services is a particular topic of debate. Table 1 shows charter schools have a similar percentage of 
students receiving special education services compared to both sectors. The impact charter schools 
have on students with English Language Learner (ELL) designation is also important to policymakers. 
The ELL student population represents three percent of all TPS students and three percent of the 
charter school population, respectively. 


® Students in poverty in this study are students identified as economically disadvantaged in the state data. In 
these data, students are either identified as economically disadvantaged or not economically disadvantaged. 
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Online charter schools have received increasing attention in the educational landscape nationally and 
in Pennsylvania. With no physical or geographic barriers to enrollment beyond state lines, online 
charter schools can draw students from across the state and use online instruction as the method of 
curriculum delivery. People often use the terms “online schools”, “cyber schools”, and “virtual schools” 
interchangeably. In this study, we use the designation of virtual schools by the National Center for 
Education Statistics (NCES). According to the definition of NCES (2016, p.9), a school is a virtual school 
if it is “a public school that only offers instruction in which students and teachers are separated by time 
or location, and interaction occurs via computers or telecommunications technologies. A virtual school 
generally does not have a physical facility that allows students to attend classes on site.” 


Table 2: Demographic Composition of Overall, Brick-and-Mortar, and Online Charter Schools: 2015-16 


Charters 

Brick-and-Mortar 

Charters 

Virtual Charters 

Number of schools 

175 

161 

14 

Average enrollment per school 

748 

604 

2,409 

Total number of students enrolled 

130,940 

97,208 

33,732 

Students in Poverty 

66% 

71% 

51% 

English Language Learners 

3% 

4% 

1% 

Special Education Students 

16% 

15% 

18% 

White Students 

35% 

23% 

69% 

Black Students 

43% 

52% 

17% 

Hispanic Students 

16% 

19% 

8% 

Asian/Pacific Islander Students 

3% 

3% 

2% 

Native American Students 

0% 

0% 

0% 

Multi-Racial Students 

4% 

3% 

4% 


As shown in a one-year snapshot in Table 2, online charter schools enroll more than 25 percent of all 
Pennsylvania charter students and serve different student populations than brick-and-mortar charters. 
Specifically, online charter schools have larger percentages of White students, smaller proportions of 
Black and Hispanic students, and fewer students living in poverty than brick-and-mortar charters. 
Students receiving special education services make up 15 percent of students attending brick-and- 
mortar charter schools while 18 percent of students in online charter schools receive special education 
services. English Language Learners constitute one percent in Pennsylvania online charters as 
compared to four percent in brick-and-mortar charters. 


“ National Center for Education Statistics (2016). Documentation to the 2014-15 Common Core of Data (CCD) 
Universe Files. Retrieved from “Public Elementary/Secondary School Universe Survey Data,” 
https://nces.ed.gov/ccd/pubschuniv.asp. 
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4. Analytic Findings of Charter School Impacts 


Overall Charter School Impact 

The primary question of this study is whether 
charter schools differ overall from traditional 
public schools in how much their students learn. 
To answer this question, we estimate the one- 
year academic gains observed for charter school 
students in each of the three studied growth 
periods and compare their average performance 
with the same measure for the VCR students. 

Please refer to the sidebar titled Graphics 
Roadmap 1 where guidance is provided to help 
readers understand the charts that follow. 

As described in the Study Approach section, 
student growth data are analyzed in units of 
standard deviations so that the results can be 
assessed for statistical differences. To help the 
reader interpret our analysis results, we 
transform standard deviation units of growth 
into days of learning based on a standard 180- 
day school year (Table 3).“ Interested readers 
can refer to the Study Approach section and 
Appendix B for detailed explanations of the 
computation of days of learning. 


Graphics Roadmap 1 

The graphics in this section have a common format. 

Each graph presents the average performance of 
charter students relative to their pertinent 
comparison students. The reference group differs 
depending on the specific comparison being made. 
Where a graph compares student subgroup 
performance, the pertinent comparison students are 
the same for both subgroups. Each graph is labeled 
with the pertinent comparison group for clarity. 

We show two axes on the graphs to help the reader get 
a sense of learning gains. The left axis indicates 
standard deviation units of learning gains of charter 
students relative to their comparison students. The 
right axis displays the same learning gains in days of 
learning. The statistical tests are performed on the 
values as they are enumerated on the left axis. 

The height of the bars in each graph reflects the 
difference in the performance between charter school 
students and the comparison student. 

Stars are used to reflect the level of statistical 
significance of the difference between the group 
represented in the bar and its comparison group of 
similar students in TPS. The absence of stars means 
that the schooling effect is not statistically different 
from zero. 


The values in Table 3 are updated from past reports using the latest (2017) NAEP scores, which show slower 
absolute annual academic progress than earlier administrations. See Eric A. Hanushek, Paul E. Peterson, and 
Ludger Woessmann, “Achievement Growth: International and U.S. State Trends in Student Performance”, 
Education Next, 12 (July 2012), 1-35. 
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Table 1: Transformation of Average Learning Gains to Days of Learning 


Standard Deviations 

Days of Learning 

0.05 

30 

0.10 

59 

0.15 

89 

0.20 

118 

0.25 

148 

0.30 

177 

0.35 

207 


Figure 3 displays the overall charter impact on student academic progress in Pennsylvania. The 
reference group, represented by the 0.00 baseline in the graph, is the average TPS VCRs in the state. 
Using the results from Figure 3 and the transformations from Table 3, we can see that in a typical school 
year, charter students in Pennsylvania experience less progress equivalent to 30 fewer days of learning 
in math in a 180-day school year. Because the difference in growth in reading is not statistically 
significant, Pennsylvania charter school students experience similar growth in the 180-day period as 
they would have in a traditional school setting. 

Figure 3: Average Learning Gains in Pennsylvania Charter Schools Compared to Average Gains for TPS VCRs 
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Charter School Impact for the 2014-2017 Cohort 

This section compares the performance of Pennsylvania charter schools between two of CREDO’S 
studies: CREDO’S 2013 study on national charter school performance^^ and this current 2019 study on 
Pennsylvania Charter Schools. Figure 4 depicts the academic growth ofthe Pennsylvania chartersector 
in the two reports. It is important to note that CREDO also released a Pennsylvania state charter school 
study in 2011^^ but there were three years of overlap between the 2011 Pennsylvania study and the 
2013 national study. The results found in these two reports closely resemble each other. 

As pointed out in the previous section, transformation of growth units of standard deviations into days 
of learning in this study is updated from past reports, using the most recent NAEP scores. Therefore, 
only growth in standard deviations is shown in Figure 4. In addition, as explained in the Study Approach 
chapter, we slightly adjust our VCR method in this study by matching a charter student by period to 
comply with the WWC Version 4.0 revised requirement for baseline equivalence. Therefore, the 
comparison ofthe overall charter effect across two reports is not purely comparable. 

Figure 4: Average Learning Gains in Pennsylvania Charter Schools Compared to Average Gains for TPS VCRs from 
the 2013 National Charter Study and 2019 Pennsylvania Study 
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^^Cremata, Edward, D. Davis, K. Dickey, K. Lawyer, Y. Negassi, M. Raymond and J. Woodworth (2013). National 
Charter School Study 2013. https://credo.stanford.edu/documents/NCSS%202013%20Final%20Draft.pdf. 

Charter School Performance in Pennsylvania, CREDO (Center for Research on Education Outcomes), Stanford 
University, April 6, 2011, http://credo.stanford.edu/reports/PA%20State%20Report_20110404_FINAL.pdf 
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Figure 4 indicates some changes in the academic progress of Pennsylvania charter school students 
between the two studies. In reading, charter students register weaker learning gains compared to their 
TPS peers in the 2013 national study, while charter school students in this study perform similarly to 
their TPS peers. In math, charter school students made less progress than their TPS VCR twins in the 
two studies, with the gap slightly smaller in the current study. 

Charter School Impact by Growth Period 

To determine whether performance is consistent over the window of this study, the impact of attending 
a charter school on academic progress is examined separately for each of the three growth periods. 
Recall that a growth period is the measure of progress from one school year to the next. In the 
presentation of results in Figure 5, the denotation "2014-2015" covers academic growth that occurred 
between the end of the 2013-2014 school year and the end of the 2014-2015 school year. Similarly, the 
denotation "2016-2017" corresponds to the year of growth between the 2015-2016 and the 2016-2017 
school years. 

Figure 5: Learning Gains in Pennsylvania Charter Schools Compared to Gains for TPS VCRs by Growth Period: 2014- 
2015 to 2016-2017 
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* Significant at p < 0.05 ** Significant at p < 0.01 

Figures reveals charter school students performing similarly to their VCR peers in readingfor all growth 
periods covered in this report. In math, however, charter school students perform significantly worse 
in the first two growth periods. In the 2014-2015 growth period, students attending charter schools 
experienced approximately 53 fewer days of learning in math, while in the 2015-2016 growth period. 
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students attending charter schools experienced approximately 30 fewer days of learning in math, 
compared to theirVCR peers. In the final growth period in our analysis, charter school students perform 
similarly to theirVCR peers in math. 

5. Charter School Analysis by School Attribute 

Charter School Impact by School Locale 

While charter schools exist in a variety of locales in the state of Pennsylvania, charter schools in urban 
areas often receive the bulk of media attention. It is important to note that charter schools in different 
locales may serve different student populations and face different obstacles related to available human 
capital. The results in Figure 6 represent the disaggregated impacts of charter school enrollment for 
urban, suburban, town, and rural charter schools. In this breakdown, charter students in different 
locations are compared with the VCR peers.For the following analysis, the comparison is relative to 
the actual progress each group of VCRs realized. But the reader should not assume that the 
transformation of each VCR group to 0.00 means that all the VCRs have equivalent academic growth. 


^''The National Center for Education Statistics (NCES) defines 12 urban-centric locales which are divided into four 
main locale types: city, suburb, rural and town. Each school’s locale is coded by NCES in the annual Core of 
Common Data dataset; we use their classification for this analysis. One important caveat is that online charter 
schools are assigned the local of their administrative office, though their students can be located in any location. 
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Figure 6: Learning Gains in Pennsylvania Charter Schools Compared to Gains for TPS VCRs by School Locale 
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Figure 6 shows urban charter students post reading growth equivalent to 35 more days of learning and 
perform similarly in math compared to their TPS counterparts. Students attending charter schools in 
the suburbs have significantly weaker growth in both subjects compared to their TPS VCRs. They 
experience about 59 fewer days of learning in reading and about 83 fewer days of learning in math. 
Students attending charter schools residing in towns experience about 124 fewer days of learning in 
both reading and math compared to their TPS peers. Students attending charter schools in rural 
settings have similar growth to their TPS peers in reading, while experiencing 47 fewer days or learning 
in math compared to their TPS peers. Refer to Appendix Table 1 for the number of student observations 
in each locale-subject combination. 

Charter School Impact by School Grade Configuration 

Charter schools often exercise their autonomy by choosing which grade levels to serve. Some charter 
operators focus on particular ages, some seek to serve a full range of grades, and others develop by 
adding one additional grade each year. The National Center for Education Statistics assigns schools the 
label of “elementary school”, “middle school”, “high school”, or “multi-level school” based on their 
predominant grade pattern. The designation of ’’multi-level school” can apply to a school that serves 
elementary and middle grades, middle and high grades, or all K-12 grades. Looking at performance 
by school grade configuration helps inform us whether specialization in a specific range of grades 


The National Center for Education Statistics (NCES) designates a school as an elementary, middle, high, or 
multi-level school. CREDO uses the designation by NCES. The sole exception is that CREDO considers a school to 
be a high school if the lowest grade served is ninth grade or above. 
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produces better results. The outcomes of students by the grade configuration of the charter school they 
attend are reported in Figure 7. Again, the comparison is relative to whatever actual progress each 
group of VCRs realized. The reader should not assume that the transformation of each VCR group to 
0.00 means that all the VCRs have equivalent academic growth. 


Figure 7: Learning Gains in Pennsylvania Charter Schools Compared to Gains for TPS VCRs by School Grade 
Configuration 
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Figure 7 shows that, on average, students in charter elementary schools experience greater growth 
than their TPS peers in reading, while performing similarly to their TPS peers in math. These students 
attending elementary charter schools experience about the equivalent of 41 more days of learning in 
reading. Students attending charter middle schools and charter high schools perform similarly in both 
reading and math compared to their VCR peers. Students attending charter multi-level schools 
experience significantly weaker growth in both reading and math compared to their TPS peers. These 
students experience about the equivalent of 41 fewer days of learning in reading and about 59 fewer 
days of learning in math. 
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6. Charter School Analysis by Student Characteristic 

Charter School Impact by Race/Ethnicity 

Since the federal government’s passage of the No Child Left Behind Act in 2001, stakeholders have used 
annual achievement tests to examine the gaps in achievement levels for students of specific racial and 
ethnic backgrounds. Part of the rationale of schools of choice such as charter schools has been the 


exploration of vehicles to lessen those 
achievement gaps. Pennsylvania’s diverse 
population as shown in Table 1 illustrates the 
need for this report to focus on the ability of 
charter schools to support the progress of 
disadvantaged students. The effectiveness of 
charter schools across ethnic and racial groups is 
especially important given the significant shares 
of historically underserved students charter 
schools enroll. 

The impacts of charter schools on the academic 
gains of Black and Hispanic students are reported 
in Figures 8 through 9a. For each student 
subgroup, we present two related graphs. 
Graphics Roadmap 2 in the sidebar provides 
guidance on how to interpret the graphs and their 
relation to each other. In short, the first graph 
depicts the growth of TPS students and charter 
students in the particular subgroup of interest as 
compared to the growth of the "average White 
TPS student". Graphs labeled “a” show whether 
the learning gains in the charter school student 
subgroup differsignificantly from theirVCRs in the 
same subgroup. 

Black students account for 43 percent of the 
charter school population in Pennsylvania. Figure 
8 shows the performance of Black students - 
regardless of their enrollment - compared to the 
performance of White students in TPS, our 
benchmark group. Black students in TPS have 
significantly weaker growth in both subjects 
compared to that of the average White student in 
TPS. Black students in TPS experience about the 


Graphics Roadmap 2 

Figures 8 through 9a show two important contrasts for 
Black and Hispanic student groups. For each student 
subgroup, we present two related graphs: 

The first graph displays the growth of TPS students 
and charter students in the particular subgroup of 

interest compared to the growth of the "average 
White TPS student." In this comparison, the White 
TPS student is male and is not in poverty, special 
education, or designated as an English Language 
Learner and is not repeating his current grade. The 
graph sets the performance of the average White TPS 
student to zero and shows how learning of students in 
the subgroup compares. 

The stars indicate the level of statistical significance. 
Thus, if there are no stars, we interpret the difference 
in learning gains as similar to the white TPS 
comparison student. The size and direction of the bars 
in the graph show the direction and magnitude of 
learning differences. If there is no difference in the 
learning gains, the bar would be missing entirely. If the 
learning of the student group in question is not as great 
as the comparison baseline, the bar is negative. If the 
learning gains exceed the comparison, the bar is 
positive. 

Graphs labeled “a” display the results of a second 
comparison testing whether the learning gains in the 
charter school student subgroup differ significantly 
from their VCRs in the same student subgroup. In 

these graphs, the performance of the TPS peers in the 
subgroup are set to zero and the learning gains of the 
charter school students in the subgroup are measured 
against that baseline. As with the first graph, stars 
denote statistical significance. 
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equivalent of 112 fewer days of learning in reading and about 106 fewer days of learning in math 
compared to the average White VCR student. Black students attending charter schools in Pennsylvania 
also exhibit weaker growth compared to the average White student in TPS, amounting to about 89 
fewer days of learning in reading and about 118 fewer days of learning in math. We refer to these 
differences as learning gaps. They have a direct impact on achievement gaps overtime. 

Figure 8: Learning Gains of Black Students in TPS and Charters Benchmarked against Learning Gains of White TPS 
Students 
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Figure 8a shows the learning gains of Black students enrolled in charter schools as compared to those 
of Black students enrolled in TPS. Black charter students experience greater yearly progress in reading 
while performing similarly in math compared to their Black TPS peers. The stronger growth in reading 
for Black charter students compared to their Black peers translates to about 24 additional days of 
learning. 

Figure 8a: Relative Learning Gains for Black Charter School Students Benchmarked against Their Black TPS Peers 
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Hispanic students account for 16 percent of the charter school population in the state of 
Pennsylvania. Figure 9 shows that Hispanic students exhibit weaker growth than the average White 
TPS student in both subjects, whether they attend TPS or charter schools. In reading, Hispanic 
students in TPS experience 83 fewer days of learning while Hispanic students in charter schools 
experience 71 fewer days of learning compared to the average White TPS student. In math, Hispanic 
students in TPS lag behind White VCR students by about 89 days, while Hispanic students in charter 
schools lag behind White TPS students by about 100 days. 

Figure 9: Learning Gains of Hispanic Students in TPS and Charters Benchmarked against Learning Gains of White 
TPS Students 


0.10 


59 


’TD 

C 

<TJ 

C 


0.05 


c 

o 


.2 0,00 


5 


-0,05 


- 0,10 


-0,15 


o 

<5 - 0.20 



30 


W) 

c 


<T3 

<U 


-30 


-59 


o 

CO 

>> 

fO 

O 


-89 


-118 




o 


-0.25 -148 

Hispanic Students in TPS Hispanic Students in Charter 

■ Reading ■ Math 

* Significant at p < 0.05 ** Significant at p < 0.01 


credo.stanford.edu 


20 


Figure 9a displays the differences in learning gains between charter school Hispanic students and 
Hispanic peers enrolled in TPS. In both subjects, Hispanic students attending charter schools in 
Pennsylvania perform similarly to their Hispanic peers attending TPS. 

Figure 9a: Relative Learning Gains for Hispanic Charter School Students Benchmarked against Their Hispanic TPS 
Peers 
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Black students and Hispanic students, regardless of the school sector they attend, both experience 
weaker growth compared to white students in TPS. Black students in charter schools outperform their 
Black peers in TPS in reading while performing similarly in math. Hispanic students in charter schools 
do not differ in growth compared to their Hispanic peers in TPS. 

Charter School Impact with Students in Poverty 

Many charter schools aim to improve educational outcomes for traditionally underserved students, 
especially for students in poverty. According to the latest data collected by the National Alliance for 
Public Charter Schools, students in poverty account for 55 percent of the national charter school 
population.^® By comparison, in Pennsylvania 66% of students attending charter schools are in poverty 
compared to 43% of TPS students. 

Figure 10 presents the annual academic growth for students in poverty. It is important to note that in 
this graph, the benchmark differs from the race/ethnicity graphs presented earlier: it is a TPS student 


The data were retrieved from “National Charter School Facts,” National Alliance for Public Charter Schools, 
https://data.publiccharters.org/ when the report was produced. 
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who is not in poverty. This analysis isolates the relationship between poverty and growth. This provides 
a picture of the difference in the impact of charter attendance on students in poverty compared to 
similar students in TPS who are not in poverty. 


Figure 10: Overall Learning Gains for TPS and Charter Students in Poverty Compared to TPS Students Not in 
Poverty 
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The results in Figure 10 illustrate differences in growth between students in poverty and TPS students 
who are not in poverty. Students in poverty attending TPS experience 65 fewer days of learning in 
reading and 59 fewer days of learning in math compared to TPS students not in poverty. Students in 
poverty attending charter schools experience 59 fewer days of learning in reading and 77 fewer days of 
learning in math compared to TPS students not in poverty. 
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Figure 10a: Relative Learning Gains for Charter School Students in Poverty Benchmarked against Their TPS Peers 
in Poverty 
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Figure 10a showsthe difference in growth between charter school students in poverty and TPS students 
in poverty. In reading, charter school students in poverty perform similarly to their TPS peers in poverty. 
In math, however, charter school students in poverty have significantly weaker growth, equivalent to 
about 18 fewer days of learning. 

Charter School Impact with Combined Race/Ethnicity and Poverty 

In public education, some of the most academically challenged students are those who are both living 
in poverty and are members of historically-underserved racial or ethnic minorities. According to the 
National Center for Education Statistics, Black and Hispanic students comprise the two race/ethnicity 
subgroups with the largest percentages of school-aged children in poverty. In 2015,36 percent of Black 
children and 31 percent of Hispanic children were living in poverty.To examine the extent to which 
performance gaps are being addressed in Pennsylvania, we further disaggregate the charter school 
impact on Black and Hispanic students in poverty. 

The impact of Pennsylvania charter schools on the academic gains of black students in poverty is 
presented in Figures 11 and 11a. The impact of charter schools on Hispanic students living in poverty is 
shown in Figures 12 and 12a. 


Kids Count Data Center | Annie E. Casey Foundation (2016). http://datacenter.kidscount.org/data/tables/44- 
children-in-povertv-bv-race-and-ethnicitv#detailed/l/anv/false/573.869.36.868.867/10.11.9.12.1.185.13/324.323 
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Figure 11: Learning Gains of Black TPS and Charter Students in Poverty Compared to Learning Gains of White TPS 
Students Not in Poverty 
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Figure 11 compares black students living in poverty, enrolled in TPS or charter schools, with the average 
White TPS student who is not in poverty. The patterns show that in Pennsylvania, black students living 
in poverty, regardless of TPS or charter enrollment, have significantly weaker growth compared to 
White TPS students who are not in poverty. Black TPS students in poverty exhibit approximately 159 
fewer days of learning in reading and 153 fewer days of learning in math than White TPS students not 
livingin poverty. Black charterstudents in poverty experience 124fewerdays of learningin readingand 
153 fewer days in math than White non-poverty TPS students. The magnitude of these results is notable. 
These students show the aggravated negative effect of their doubly disadvantaged status. 
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Figure 11a: Relative Learning Gains for Black Charter School Students in Poverty Benchmarked against Their 
Black TPS Peers in Poverty 
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When comparing black students in poverty attending charter schools to black students in poverty 
attending TPS, there is a positive charter effect in reading and no difference in the math growth. The 
results for this comparison are presented in Figure 11a. In reading, black students in poverty attending 
charter schools gain about 35 additional days of learning compared to their black TPS peers in poverty. 
In math, there is no difference in the average growth between the two groups. 
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Figure 12: Learning Gains of Hispanic TPS and Charter Students in Poverty Compared to Learning Gains of White 
TPS Students Not in Poverty 
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As shown in Figure 12, Hispanic students in poverty, regardless of enrollment in charter schools or TPS, 
have significantly weaker growth in both reading and math compared to White non-poverty students 
in TPS. Hispanic TPS students in poverty lag behind their White non-poverty TPS peers at a rate 
equivalent to 130 less days of learning in both reading and math. Hispanic Charter students in poverty 
experience weaker gains at the rate of 100 fewer days of learning in reading and 136 fewer days or 
learning in math comparted to White students in TPS. 
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Figure 12a: Relative Learning Gains for Hispanic Charter School Students in Poverty Benchmarked against Their 
Hispanic TPS Peers in Poverty 


0.10 

0.05 


.1 0.00 


I -0.05 


■g -0,10 

4 -» 

(/) 

C -0.15 


I. 


2 

o 


0.20 


-0.25 

-0.30 


0.04“ 


■ Reading 

' Significant at p< 0.05 


0.00 


Math 

“ Significant at p< 0.01 


59 


30 


-30 


bo 

_c 

'c 

<13 


o 

<rt 


-59 >1 

(13 

Q 


-89 


-118 


-148 


-177 




Figure 12a shows that Hispanic students in poverty attending charter schools gain about 24 days of 
learning in reading and have similar growth in math when compared to Hispanic students in poverty 
attending TPS. 

Charter School Impact with Special Education Students 

In Pennsylvania, 15 percent of TPS students and 16 percent of charter school students receive special 
education services. Ideally, we would compare outcomes for each Individual Education Program (lEP) 
designation. Unfortunately, that approach is not feasible due to the large number of categories and the 
relatively small number of students in each. Faced with this challenge, we aggregate across all 
categories of special education. Therefore, the results of this section should be interpreted with 
caution. 
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Figure 13: Learning Gains for TPS and Charter Students in Special Education Compared to TPS Students Not in 
Special Education 
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in Figure 13, we benchmark TPS students receiving special education services and charter students 
receiving special education services to the TPS VCRs who do not receive special education services. 
Students in special education attending Pennsylvania public schools, regardless of sector, experience 
weaker growth compared to their non-special education TPS peers. Students in special education 
attending a TPS experience about the equivalent of 89 fewer days of learning in reading and about 47 
fewer days of learning in math compared to TPS students who are not in special education. Special 
education students attending charter schools experience about 106 fewer days of learning in reading 
and about 83 fewer days of learning in math compared to their TPS students not receiving special 
education services. 
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Figure 13a: Relative Learning Gains for Charter Students in Special Education Benchmarked against Their TPS 
Peers in Special Education 
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Figure 13a represents the growth of special education students attending charter schools compared to 
special education students attending TPS. Charter school students receivingspecial education services 
have significantly weaker growth compared to their TPS peers in both reading and math. Charter 
special education students experience about 24 fewer days of learning in reading and about 35 fewer 
days of learning in math compared to their TPS peers also receiving special education services. 

Charter School Impact with English Language Learners 

The 2017 National Assessment of Education Progress (NAEP) documented the performance gap 
between English Language Learners (ELL) and their English-proficient peers, showing ELL students 
have weaker performance.^® The analyses presented in Figure 14 and Figure 14a highlight differences 
in ELL students in charter schools and ELL students in TPS. 


“ The Nation’s Report Card (2018). 2017 Mathematics and Reading Assessments 
https://www.nationsreportcard.gov/math_2017/nation/gaps/?grade=4#?grade=4. 
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Figure 14: Learning Gains for TPS and Charter ELL Students Compared to TPS Non-ELL Students 
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Figure 14 shows that regardless of sector, English Language Learners in Pennsylvania public schools 
exhibit weaker growth in both reading and math compared to TPS students who are English proficient. 
ELL students enrolled in TPS have about 53 fewer days of learning in reading and about 30 fewer days 
of learning in math compared to TPS English-proficient students. ELL Students in charter schools have 
about the equivalent of 41 fewer days of learning in reading and 30 fewer days of learning in math 
compared to English-proficient students attending TPS. Figure 14a below shows that charter school 
students with ELL designation and TPS students with ELL designation have similar growth in both 
reading and math. 
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Figure 14a: Relative Learning Gains for Charter ELL Students Benchmarked against Their TPS ELL Peers 
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Table 4 summarizes the effect that charter schools have on different student groups. The results 
represent the growth of each group relative to their peer group in TPS. Black students in charter 
schools, for example, experience additional reading growth of .04 (24 days) compared to Black students 
in TPS. 


Table 4: Charter School Impacts on Student Subgroup Learning Gains 


Student Group 

Charter School Effect on Student Groups 
Benchmarked against their TPS Peers 


Reading 

Math 

Overall Charter School Effect 

-0.02 

-0.05** 


Charter School Students in Poverty 

0.01 

-0.03* 

Black Charter Students 

0.04* 

-0.02 

Black Charter Students in Poverty 

0.06** 

0.00 

Hispanic Charter Students 

0.02 

-0.03 

Hispanic Charter Students in Poverty 

0.04** 

0.00 

Special Education Charter Students 

-0.04* 

-0.06** 


English Language Learner Charter Students 


- 0.02 


- 0.01 
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Charter School Impact by Students’ Years of Enrollment 

The academic growth of a students attending charter schools may change along with the number of 
years a student is enrolled in a charter school. To test the relationship between progress and the length 
of enrollment in a charter school, we separately group test scores from students in the first year of 
charter enrollment, scores from students in their second year of charter attendance, and scores of 
students in their third year in a charter school. In this scenario, the analysis is limited to the charter 
students who enroll for the first time in a charter school between the 2014-15 and 2016-17 school years 
and their TPS VCRs. Although this approach reduces the number of students included, it ensures an 
accurate measure of the effect of continued enrollment overtime. The results for this subset of the full 
study sample should not be directly compared with other findings in this report. Figure 15 shows the 
results. 

Figure 15: Learning Gains of Charter Students Compared to Gains for Average TPS VCRs by Students’ Years of 
Enrollment in Charter Schools 
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Figure 15 shows that students who are in theirfirstyear of charter school enrollment have significantly 
weaker growth compared to the average TPS VCRs. Students in their first year of charter school 
enrollment experience about 77 fewer days of learning in reading and about 100 fewer days of learning 
in math compared to the average TPS student. Students in their second or third year of charter school 
enrollment however perform similarly to the average TPS student in both subjects. 
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7. School-level Analysis 

The numbers reported in the previous sections represent the typical learning gains at the student level 
across the state; they reveal what would be the likely result if a typical student enrolled in any of the 
Pennsylvania charter schools. The results pool all students in all schools in all growth periods. The prior 
results do not let us discern if some charter schools are better than others. Since school-level results 
are of interest to policy makers, parents and the general public, we aggregate charter student 
performance to the school level for each charter school in the state. This view is necessarily limited to 
charter schools with sufficient number of tested students to make a reliable inference on performance. 

Comparative School-Level Quality 

It is important to understand the counterfactual used in this section. As shown in Table 1 earlier in the 
report, the student populations within the typical charter school and their feeder schools differ, making 
whole-school to whole-school comparisons unhelpful. Here instead, we pool each school’s VCRs to 
simulate the “apples to apples” TPS to serve as the control condition for testing the performance of 
charter schools. This simulated TPS reflects a precise estimate of the alternative local option for the 
students actually enrolled in each charter school. 

To determine the range of charter school performance, we estimate the annual learning impact of each 
Pennsylvania charter school over the two most recent growth periods (2015-2016 and 2016-2017).^® The 
estimated learning impact for each charter school can be positive (statistically different from zero with 
a positive sign), negative (statistically different from zero with a negative sign), or zero. We use it to infer 
how the academic quality of a charter school compares to the quality of traditional public schools 
which students in that charter school would have potentially attended if they had not attended a 
charter school. 

A statistically positive learning impact for a charter school suggests that the charter school has stronger 
learning growth than the alternative TPS options for its students. A statistically negative learning 
impact for a charter school implies the school makes less progress than the traditional schools its 
students would have attended. A zero learning impact means that the charter school and the TPS 
alternatives for its students have similar performance. 

Our total sample consists of 155 schools with reading scores and 152 schools with math scores in the 
2015-2016 and 2016-2017 growth periods. Table 5 below shows the breakout of the performance for 
the included Pennsylvania charter schools. 


We chose to include only the two most recent growth periods in this analysis in consideration of the dynamic 
growth within some charter schools and to provide the most contemporary picture of performance possible. 

As noted in Table 1, charter schools are smaller on average than their corresponding feeder schools. 
Furthermore, some charter schools elect to open with a single grade and mature one grade at a time. 
Consequently, care is needed when making school-level comparisons to ensure that the number of tested 
students in a school is sufficient to provide a fair representation of the school’s impact. Our criterion for 
inclusion is at least 60 matched charter student records over the two growth periods or at least 30 matched 
charter records for schools with only one growth period. 
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Table 5: Performance of Charter Schools Compared to Their TPS Alternatives in Pennsylvania 


Subject 


Significantly Worse Not Significantly Different Significantly Better 
Number Percent Number Percent Number Percent 



35 


23% 


51 


33% 


69 


45% 



48 


32% 


54 


36% 


50 


33% 


In reading, 45 percent of charter schools perform significantly better than the traditional schooling 
environments the charter students would have otherwise attended. In math, 33 percent perform 
significantly better than TPS alternatives. To benchmark these figures nationally, 25 percent of charter 
schools in the nation outperform their local counterparts in reading and 29 percent do so in math.^^ At 
the other side of the distribution, 23 percent of Pennsylvania charter schools have significantly weaker 
reading results than their local TPS counterparts, and 32 percent do so in math. Comparing to the 
national picture, 19 percent of charter schools in the nation pale against the local TPS alternatives in 
reading and 31 percent do so in math. In reading, 33 percent of Pennsylvania charter schools have 
results that do not differ significantly from the local TPS option. In math, 36 percent of Pennsylvania 
charter schools have similar growth performance compared to the local TPS alternatives. 


CREDO (2013). National Charter School Study 2013. http://credo.stanford.edu. 
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Growth and Achievement 

The impacts of charter schools on academic 
growth relative to the local competition are 
informative for many questions, but they do 
not indicate how well students perform in 
absolute terms. Since many of the students 
served by charter schools start at low levels of 
achievement, the combination of absolute 
achievement and relative growth is vital to 
understanding their success. 

For each school, the tested achievement of its 
students over the same two periods covered by 
the academic growth analysis (2015-2016 and 
2016-2017) is averaged and transformed to a 
percentile within the statewide distribution of 
achievement.^ The 50th percentile indicates 
statewide average performance for all public 
school students (traditional and charter). A 
school achievement level above the 50th 
percentile indicates that the school's overall 
achievement exceeds the statewide average. 
We use the effect sizes discussed above to 
measure growth. We display each school’s 
achievement and growth effect size in a two- 
dimensional plot, displayed in Tables 6 and 7. 


Graphics Roadmap 3 

Here is a note about how to interpret the results in 
Tables 6 and 7: 

There are four quadrants in each table. We have 
expanded on the usual quadrant analysis by dividing 
each quadrant into four sections. The value in each 
box is the percentage of charter schools with the 
corresponding combination of growth and 
achievement. The value in the center of each quadrant 
is the sum of the four sections in that quadrant. These 
percentages are generated from the 2016 and 2017 
growth periods. 

The uppermost box on the left denotes the percentage 
of charters with very low average growth but high 
average achievement. The box in the bottom left 
corner depicts low-growth, low-achieving schools. 

Similarly, the uppermost box on the right contains the 
percentage of charters with high average growth and 
high average achievement. The bottom right corner 
contains high-growth, low-achieving schools. 

The major quadrants were delineated using national 
charter school data. We would expect the majority of 
schools to have an effect size between -0.15 and 0.15 
standard deviations of growth (the two middle 
columns). Similarly, we would expect about 40 
percent of schools to achieve between the 30**’ and 
70**’ percentiles. These expectations are based on how 
we view a normal distribution with the majority of the 
sample falling within one standard deviation of the 
mean. 


Average achievement was computed using students’ z-scores from the end of the growth period (e.g., spring 
2016 and spring 2017), and the resulting school-level mean was then converted into a percentile. 
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Table 6: School-Level Reading Growth and Achievement in Pennsylvania Charter Schools 


Low Growth, High Growth, 

High Achievement High Achievement 



70th Percentile 


50th Percentile 


30th Percentile 


Low Growth, High Growth, 

Low Achievement Low Achievement 


Table 6 shows the reading achievement and growth results for the Pennsylvania charter schools 
included in this analysis. In reading, 97 out of 155 Pennsylvania charter schools (62.4 percent) have 
positive average growth (this percentage is the sum of the eight squares in the blue and pink quadrants 
on the right half of the table). Only 13 percent of charters have positive growth and average 
achievement above the 50‘*' percentile (i.e., the total for the blue quadrant on the top right). A total of 
49 percent of charter schools in the pink box post above average reading growth, while posting below 
average achievement. 

About 37 percent of schools post lower than average growth (the sum of gray and brown quadrants on 
the left half of the table). Approximately 81 percent of Pennsylvania charter schools perform below the 
50‘^ percentile in achievement (the sum of the brown and pink cells in the lower half of the table). The 
area of the greatest concern is the 32 percent of Pennsylvania charter schools that lie in the lower left 
quadrant in the table. These schools are characterized by both low achievement and low growth in 
reading. 
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Table 7: School-Level Math Growth and Achievement in Pennsylvania Charter Schools 


Low Growth, High Growth, 

High Achievement High Achievement 



70th Percentile 


50th Percentile 


30th Percentile 


Low Growth, High Growth, 

Low Achievement Low Achievement 


In math, 85 of the 152 Pennsylvania charter schools (around 56 percent) have positive average growth 
in math, as seen in the combined orange and pink quadrants on the right half of Table 7. Just under 10 
percent of Pennsylvania charter school exhibit stronger than average growth, and post achievement 
above the 50**' percentile (the orange quadrant in the upper right of the table). Almost 87 percent of 
charter schools in Pennsylvania post below-average achievement (sum of the cells in the lower half of 
the table). As in the previous table, the schools of the greatest concern are those schools in the lower 
left (brown) quadrant that demonstrate both low achievement and low growth; they account for 77 
schools (nearly 41 percent) of the charter schools in Pennsylvania. 
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8. Analysis of Online Charter Schools 

Overall Charter Impact by Delivery System 

Pennsylvania consists of both brick-and-mortar charter schools and online charter schools. Students 
from all over the state can attend online charter schools and receive instruction online. As Table 2 
revealed, online charter schools enroll over 25 percent of charter students in Pennsylvania and have 
different student compositions compared to brick-and-mortar charters. CREDO’S earlier study on 
online charter schools also found that online charter schools serve students with higher mobility rates 
and have significantly negative impacts on student academic progress.^^ 

In this section, we break down the charter school impact on student performance by delivery system. 
Figure 16 compares the performance of students in online charter schools and students in brick-and- 
mortar charters to the performance of the "average TPS VCR." This is followed by Figure 16a that 
displays the difference in learning of students enrolled in online charter schools compared to those 
who attend brick-and-mortar charters. 


Woodworth, J., Raymond, M., Chirbas, K., Gonzalez, M., Negassi, Y., Snow, W., VanDonge, C. Online Charter 
School Study (2015). https://credo.stanford.edu/pdfs/Online%20Charter%20Study%20Final.pdf. 
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Figure 16: Student Learning Gains for Students in Pennsylvania Online and Brick-and-Mortar Charter Schools 
Benchmarked against Learning Gains for Average TPS VCRs 
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Figure 16 shows that students attending online charter schools have weaker growth in both reading 
and math compared to the average TPS VCR. These gaps translate to 106 fewer days of learning in 
reading and 118 fewer days of learning in math. Students attending brick-and-mortar charter schools 
however exhibit positive growth in reading compared to the average TPS VCR, gaining about 24 days of 
learning. In math, brick-and-mortar charter school students perform similarly to the average TPS VCR. 

Figure 16a compares the performance of students attending online charter schools against that of 
students attending brick-and-mortar charter schools (whose baseline is represented by the 0.00 line). 
Online charter school students have significantly weaker growth in both subjects. Students attending 
online charter schools experience about the equivalent of 130 fewer days of learning in reading and 
about 118 fewer days of learning in math compared to students attending brick-and-mortar charter 
schools. 
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Figure 16a: Student Learning Gains in Pennsylvania Online Charter Schools Benchmarked against Students in 
Pennsylvania Brick-and-Mortar Charter Schools 
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Two important points arise from these graphics. First, Pennsylvania students in online charter schools 
have learning gains in both subjects that dramatically lag behind both the average TPS and brick-and- 
mortar charter school peers. Second, the negative overall charter impact on math progress in Figure 3 
is driven by the strikingly negative math growth of students in online charter schools. The performance 
of the online schools depresses the overall sector performance despite serving only one quarter of 
charter school students in the state. 
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Online Charter School Impact for the 2014-2017 Cohort 

This section compares the performances of Pennsylvania online charter schools across two of CREDO’S 
studies: CREDO’S 2015 Online Charter School Study and this current 2019 study on Pennsylvania charter 
schools. Figure 17 depicts the academic growth of Pennsylvania’s online charter sector in the two 
reports. It is important to reiterate that the transformations of growth units of standard deviations into 
days of learning in this study is updated from past reports, using the most recent NAEP scores. 
Therefore, only growth in standard deviations is shown in Figure 17. In addition, as explained in the 
Study Approach chapter, we tweak our VCR method a little in this study by matching a charter student 
by period as to meet the WWC Version 4.0 requirement for baseline equivalence. Therefore, the 
comparison of the overall online charter school effect across these three reports is only suggestive. 

Figure 17: Average Learning Gains in Pennsylvania Online Charter Schools Compared to Average Gains for TPS 
VCRs from the 2015 Online Study and 2019 Pennsylvania Study 
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Figure 17 shows little change in the academic progress of Pennsylvania online charter school students 
across the studies. Online charter schools register weaker learning gains in both reading and math 
compared to the average TPS VCR across both studies. This study reveals a slightly larger reading gap 
between online charter school students and the average VCR compared to the 2015 Online Study. In 
math, the gaps have slightly shrunk. Overall, this graph shows no substantial academic improvement 
of Pennsylvania’s online charter sector across these two studies. 
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Table 8 summarizes the effect that online and brick-and-mortar charter schools have on different 
student groups. The results represent the growth of each group relative to TPS VCRs in the same 
subgroup. The growth data for brick-and-mortar charters and online charters found in the rows labeled 
“Overall Impact” correspond to those in Figure 16. 

Table 8: Learning Gains of Online and Brick-and-Mortar Charter School Student Subgroups Compared to Gains 
of TPS VCRs in the Same Subgroup 


Student Group 

Overall Charter 

Brick-and-Mortar 

Charter 

Online Charter 

1 Reading 1 

Overall Impact 

-0.02 

0.04** 

-0.18** 

Special Education Students 

-0.04* 

0.01 

-0.16** 

Students in Poverty 

0.01 

0.05** 

-0.16** 

Black Students 

0.04* 

0.06** 

-0.08** 

Hispanic Students 

0.02 

0.05** 

-0.12** 

liviath 1 

Overall Impact 

-0.05** 

-0.01 

-0.20** 

Special Education Students 

-0.06** 

-0.01 

-0.17** 

Students in Poverty 

-0.03* 

0.00 

-0.19** 

Black Students 

0.02 

-0.01 

-0.11** 

Hispanic Students 

-0.03 

0.00 

-0.15** 


In reading, the overall positive impact of brick-and-mortar charter schools in Pennsylvania holds for 
students in poverty, Black students, and Hispanic students. Students in brick-and-mortar charter 
schools with these characteristics gain up to 35 more days of learning in reading compared to TPS VCRs 
in the same subgroup. Students receiving special education services in brick-and-mortar charter 
schools perform similarly in reading compared to TPS VCRs in special education. In math, we observe 
similar performance of brick-and-mortar charter school special education students, students in 
poverty. Black students, and Hispanic students in comparison with TPS VCRs in the same subgroup. 
The overall negative impact of online charter schools relative to TPS VCRs is also found for special 
education students, students in poverty. Black students, and Hispanic students in both reading and 
math. 

We can draw two main conclusions from Table 8. The first is that the disaggregation of the charter 
impact of subgroups into the two subsectors shows that online charter schools depress the charter 
effect for each of these subgroups. We can also conclude that the negative impact for online charter 
schools cannot be attributed to their demographic composition, since the negative impact is found 
across many subgroups. 
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9. Synthesis and Conclusions 

Summary of Major Findings 

This study examines the academic progress of students in Pennsylvania charter schools over a four- 
year period. Our data window ranges from the 2013-2014 school year to the 2016-2017 school year, used 
to create three year-to-year measures of progress, or “growth periods”. Table 9 presents a summary of 
the results from the various analyses in this report. 

Table 9: Summary of Analysis Findings for Pennsylvania Charter School Students Benchmarked against 


Comparable TPS Students 



Reading 

Math 

Pennsylvania Charter Students 

Similar 

Negative 

Students in Charters in 2014-2015 

Similar 

Negative 

Students in Charters in 2015-2016 

Similar 

Negative 

Students in Charters in 2016-2017 

Similar 

Similar 

Students in Urban Charter Schools 

Positive 

Similar 

Students in Suburban Charter Schools 

Negative 

Negative 

Students in Town Charter Schools 

Negative 

Negative 

Students in Rural Charter Schools 

Similar 

Negative 

Students in Elementary Charter Schools 

Positive 

Similar 

Students in Middle School Charter Schools 

Similar 

Similar 

Students in High School Charter Schools 

Similar 

Similar 

Students in Multi-level School Charter Schools 

Similar 

Negative 

Black Charter School Students 

Negative 

Negative 

Hispanic Charter School Students 

Negative 

Negative 

Charter School Students in Poverty 

Negative 

Negative 

Black Charter School Students in Poverty 

Negative 

Negative 

Hispanic Charter School Students in Poverty 

Negative 

Negative 

Special Education Charter School Students 

Negative 

Negative 

English Language Learner Charter School Students 

Negative 

Negative 

Students in First Year Enrolled in Charter School 

Negative 

Negative 

Students in Second Year Enrolled in Charter School 

Similar 

Similar 

Students in Third Year Enrolled in Charter School 

Similar 

Similar 

Students in Online Charter Schools 

Negative 

Negative 

Students in Brick-and-Mortar Charter Schools 

Positive 

Similar 

Special Education Students in Online Charter Schools 

Negative 

Negative 

Special Education Students in Brick-and-Mortar Charter Schools 

Similar 

Similar 

Students in Poverty Attending Online Charter Schools 

Negative 

Negative 

Students in Poverty Attending Brick-and-Mortar Charter Schools 

Positive 

Similar 

Black Students in Online Charter Schools 

Negative 

Negative 

Black Students in Brick-and-Mortar Charter Schools 

Positive 

Similar 

Hispanic Students in Online Charter Schools 

Negative 

Negative 

Hispanic Students in Brick-and-Mortar Charter Schools 

Positive 

Similar 
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Over the three growth periods in this study, the typical charter school student in Pennsylvania had 
similar academic growth in reading and weaker math growth compared to their TPS counterparts. In 
math, the learning difference is about the equivalent to losing 30 days of learning compared to their 
TPS peers. In the first two growth periods of the study, students in Pennsylvania charter schools 
experience growth similar to their TPS peers in reading, while experiencing weaker growth in math. By 
the third growth period, students in Pennsylvania charter schools exhibit similar growth to their TPS 
counterparts in both reading and math. 

Beyond the overall findings, the analysis provides more nuanced insight into charter school 
performance in Pennsylvania. Students attending urban charter schools learn more than their TPS 
peers in reading while performing similarly in math. Students attending rural charter schools perform 
similarly to their TPS peers in reading while showing significantly weaker growth in math. Students 
attending charter schools in suburban or town settings have significantly weaker growth compared to 
their TPS peers in both subjects. 

The analysis by school grade configuration shows that students in elementary school experience 
greater growth than their TPS peers in reading, while performing similarly to their TPS peers in math. 
Students in multi-level charter schools have significantly weaker growth in both reading and math 
compared to their peers in TPS. Students in charter middle schools or charter high schools have similar 
growth to their TPS peers in both reading and math. 

Black students attending charter schools in Pennsylvania gain approximately 24 additional days of 
reading than their Black TPS peers, while having similar growth in math. Hispanic students attending 
Pennsylvania charter schools havesimilargrowth to their Hispanic TPS peers in both reading and math. 
Charter school students in poverty post similar growth to their TPS peers in reading while posting 
significantly weaker growth to their TPS peers in math, translating to 18 fewer days of learning. 
However, Black students in poverty post significantly stronger reading growth than their Black TPS 
peers in poverty, with gains of approximately 35 additional days of learning, while having similar 
growth in math. Hispanic students in poverty also exhibit stronger growth in readingthan their Hispanic 
TPS peers in poverty (approximately 24 additional days of learning), while posting similar growth in 
math. 

Charter school students receiving special education services have significantly weaker growth than 
their TPS peers in both reading and math. These results are realized as 24 fewer days of learning in 
reading and 35 fewer days of learning in math. Charter school students with ELL designation perform 
similarly to their TPS peers with ELL designation in both reading and math. 

We also find that on average, students in their first year of charter school enrollment post significantly 
weaker growth than their TPS peers, losing the equivalent of about 77 days of learning in reading and 
about 100 days of learning in math. Students in their second or third year of enrollment post similar 
growth to the TPS peers. 
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The section of our analysis regarding charter impact at the school level reveals that almost half of the 
charter schools in Pennsylvania outpace their local TPS counterparts in reading, while one third of 
charter schools outpace their local TPS counterparts in math. About one third of schools perform 
similarly to their TPS counterparts in both reading and math. Almost one quarter of schools 
underperform compared to the TPS counterparts in reading and one third underperform compared to 
the TPS counterparts in math. 

Our school level analysis also reveals that 81 percent of Pennsylvania charter schools perform below 
the SO**" percentile in reading achievement and almost 87 percent of Pennsylvania charter schools 
perform below the SO**" percentile in math achievement. Locational decisions and the starting points of 
the students they serve influence these outcomes. In addition, 49 percent of all charter schools post 
lower-than-average achievement and above-average growth outcomes in reading, while 32 percent of 
charter schools post both below-average achievement and below-average growth. In math, 46 percent 
of charter schools post lower-than-average achievement with above-average growth, while almost 41 
percent of charter schools post below- average achievement and below-average growth. 

The debate over online charter schools is of particular interest in Pennsylvania. Our analysis reveals 
that students enrolled in Pennsylvania online charter schools post significantly weaker growth than the 
average TPS student and the average brick-and-mortar charter school student. Compared to the 
average TPS student, a student enrolled in an online charter school loses the equivalent of 106 days of 
learning in reading and about 118 days of learning in math. Students attending brick-and-mortar 
charter schools show significantly stronger growth in reading by 83 additional days of learning and 
similar growth in math compared to the average TPS VCR. 

Our online school analysis also includes the disaggregation of results by special education status, 
poverty status, and race. The poor results for online charter school students are consistent across these 
student subgroups, so this effect is not being driven by student composition in online charter schools. 
It is also important to note that positive charter results in reading hold for students attending brick- 
and-mortar charter schools across many of the same subgroups. Students in poverty. Black students, 
and Hispanic students in brick-and-mortar charter schools all post stronger reading gains compared to 
the average TPS VCR. With this deeper analysis of online charter schools in Pennsylvania, we conclude 
that the poor performance of online charter schools is responsible for the overall low performance of 
the sector in the combined analyses. 
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Implications 

Overall, there has been little to no progress in Pennsylvania charter school performance since CREDO’S 
2013 National Charter School Study. 

Despite the generally flat performance overall, there are important positive findings for Pennsylvania 
charter schools. Most notable is the strong reading performance of brick-and-mortar set of charter 
schools. The impact on student academic growth from the brick-and-mortar charter subsector in 
Pennsylvania is on par with recognized strong charter sectors in other states such as Indiana, Illinois, 
or North Carolina. There are many schools that serve as strong examples in both reading and math 
performance that could provide valuable models for all Pennsylvania schools to emulate. Of special 
interest are the schools that show high growth and high achievement. Knowledge transfer and 
supported replication of these school models could rapidly increase the number of high quality seats 
in Pennsylvania. 

At the same time, the evidence shows that Pennsylvania has substantial numbers of underperforming 
charter schools. To be clear, the proportion of sub-par charter schools has declined since our 2011 
Pennsylvania study. However, with nearly one quarter of the schools lagging in reading and one third 
in math, the collective impact on student’s academic careers and later life outcomes remains of deep 
concern. 

This report found overwhelmingly negative results found from online charter schools; any potential 
benefits of online schooling such as student mobility and flexibility in curriculum are drowned out by 
the negative impacts on academic growth of students enrolled in such schools. Urgent attention of two 
forms is needed. First, education leaders must assess the experience of the specific students enrolled 
in the online schools to ascertain if their educational needs are being met in their current schools. 
Second, policy makers need to determine if current oversight policies or practices for online charter 
schools are sufficient to assure adequate performance. 
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Appendix A. Number of Observations for All Results 

The numbers in the table below represent the number of charter observations associated with the 
corresponding results in the report. An equal number of VCRs were included in each analysis. 


Appendix Table 1: Number of Observations for All Results 

Student Group 

Matched Charter Student Records 


Reading 

Math 


Pennsylvania Charter Students Tested & Matched 

122,735 

123,759 

Students in Charters in 2012-2013 

38,581 

40,682 

Students in Charters in 2013-2014 

38,628 

39,347 

Students in Charters in 2014-2015 

45,526 

43,730 

Students in Urban Charter Schools 

70,515 

71,347 

Students in Suburban Charter Schools 

40,903 

41,018 

Students in Rural Charter Schools 

9,113 

9,194 

Students in Town Charter Schools 

2,204 

2,200 

Students in Elementary Charter Schools 

40,234 

40,408 

Students in Middle School Charter Schools 

8,900 

9,033 

Students in High School Charter Schools 

1,499 

1,458 

Students in Multi-level School Charter Schools 

72,102 

72,860 

Students in First Year Enrolled in Charter School 

24,046 

24,225 

Students in Second Year Enrolled in Charter School 

7,755 

7,985 

Students in Third Year Enrolled in Charter School 

2,499 

2,548 

Black Charter School Students 

57,117 

57,846 

Hispanic Charter School Students 

19,189 

19,656 

White Charter School Students 

40,140 

40,074 

Charter School Students in Poverty 

86,289 

87,374 

Black Charter School Students in Poverty 

48,934 

49,604 

Hispanic Charter School Students in Poverty 

16,357 

16,793 

Special Education Charter School Students 

19,376 

20,062 

English Language Learner Charter School Students 

2,715 

2,951 

Grade Repeating Charter School Students 

1,189 

1,330 

Online Charter School Students 

30,059 

30,334 

Brick-and-Mortar Charter School Students 

92,649 

93,403 

Special Education Students in Online Charter Schools 

5,662 

5,753 

Special Education Students in Brick-and-Mortar Charter Schools 

13,713 

14,305 

Students in Poverty Attending Online Charter Schools 

14,866 

15,046 

Students in Poverty Attending Brick-and-Mortar Charter Schools 

71,404 

72,309 

Black Students in Online Charter Schools 

5,159 

5,255 

Black Students in Brick-and-Mortar Charter Schools 

51,943 

52,570 

Hispanic Students in Online Charter Schools 

2,383 

2,437 

Hispanic Students in Brick-and-Mortar Charter Schools 

16,798 

17,215 
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Appendix B. Technical Appendix 


Match Rates for Pennsylvania Charter Students with Feeder List Restricted and Not Restricted 

In the Study Approach chapter, we explain that the United States Department of Agriculture phased in 
the Community Eligibility Provision (CEP) in Pennsylvania and otherstatesduringthe study period. The 
CEP allows schools and local education agencies with a minimum Identified Student Percentage (40 
percent or more) to provide free breakfast and lunch to all students. To minimize over-identification of 
students living in poverty in the analysis, we drop from the list of feeder schools a very small number of 
TPS if their share of the students identified as economically disadvantaged by the state was 100 percent 
and represented a jump by 35 percentage points or more from the previous year. As Appendix Table 2 
shows, restricting the feeder list did not affect the percentage of charter students for whom a VCR 
match was possible. 


Appendix Table 2: Match Rates for Tested Charter School Student Observations in Pennsylvania with Feeder 
List Restricted and Not Restricted 



Reading Math 

Match Rate With Full Feeder List 

Match Rate With Feeder List Restricted 

84% 84% 

84% 84% 


Note: The feeder list with restriction does not include a very small number of TPS feeders whose share of 
economically disadvantaged students was 100 percent and represented a jump by 35 percentage points or more 
from the previous year. 

Demographic Composition of Charter Students in the Study 

This study examines the performance of students in charter schools who participated in annual 
accountability testing in Pennsylvania, occurring in grades 3-8 and in whatever grade the end-of-course 
(EOC) assessments were taken. The test scores allow us to use a common measure of performance 
across schools and over time. However, in each growth period of the study, students who are enrolled 
in non-tested grades are not included in the analysis of performance. This partially accounts for the 
differences in school and student counts in our analysis data compared to other published figures 
about the charter school population in Pennsylvania. 


credo.stanford.edu 


48 






As discussed in the Study Approach chapter, we match tested charter students by period if they can be 
tracked for two or three periods in the study to conform to the new baseline equivalence requirement 
in the Procedures Handbook Version 4.0 of What Works Clearinghouse. The following three tables 
present the student profiles of all and matched tested charter students in math in Pennsylvania in each 
matching period. 

Appendix Table 3: Demographic Composition of Charter Students in the Study: Period 1 

Student Group 

All Charter Students Tested 

Matched Charter Students 

Number I 

1 Percent I 

1 Number I 

1 Percent I 

Pennsylvania Charter Students 

86,416 


72,441 


% Matched 

84% 




Black Students 

37,888 

44% 

32,155 

44% 

Hispanic Students 

14,381 

17% 

11,397 

16% 

White Students 

28,353 

33% 

25,203 

35% 

Students in Poverty 

59,000 

68% 

49,771 

69% 

Special Education Students 

15,389 

18% 

11,723 

16% 

English Language Learners 

2,684 

3% 

1,762 

2% 

Grade Repeating Students 

1,717 

2% 

771 

1% 

Appendix Table 4: Demographic Composition of Charter Students in the Study: Period 2 


Student Group 

All Charter Students Tested 

Matched Charter Students 






Number 

Percent 

Number 

Percent 

Pennsylvania Charter Students 

41,894 


35,377 


% Matched 

84% 




Black Students 

19,767 

47% 

17,579 

50% 

Hispanic Students 

7,088 

17% 

5,642 

16% 

White Students 

12,127 

29% 

10,465 

30% 

Students in Poverty 

29,653 

71% 

25,785 

73% 

Special Education Students 

7,461 

18% 

5,760 

16% 

English Language Learners 

1,227 

3% 

826 

2% 

Grade Repeating Students 

833 

2% 

405 

1% 

Appendix Table 5: Demographic Composition of Charter Students in the Study: Period 3 


Student Group 

All Charter Students Tested 

Matched Charter Students 

Number 

Percent 

Number 

Percent 

Pennsylvania Charter Students 

18,893 


15,947 


% Matched 

84% 




Black Students 

9,192 

49% 

8,091 

51% 

Hispanic Students 

3,170 

17% 

2,613 

16% 

White Students 

5,181 

27% 

4,410 

28% 

Students in Poverty 

13,500 

71% 

11,799 

74% 

Special Education Students 

3,387 

18% 

2,575 

16% 

English Language Learners 

526 

3% 

363 

2% 

Grade Repeating Students 

329 

2% 

151 

1% 
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Comparison of Starting Scores of Matched Students and VCRs 

The VCR method used in this study of Pennsylvania provided matches for 84 percent of tested charter 
students with growth scores in both reading and math. To assess the quality of the matches, we 
compare the starting scores of matched charter students and the Virtual Control Records obtained from 
the matches in both reading and math. The statistical tests of equality of means are shown in Appendix 
Figures 1 and 2 for math and reading, respectively. We find that the starting scores of matched students 
and the “virtual twins” used as point of comparison are almost identical. As matched students and their 
“virtual twins” have identical starting points in terms of learning in the beginning of a growth period, 
we can be confident that any difference in their final scores and therefore their learning growth can be 
attributed to charter school attendance, as the only observed way in which matched students and VCRs 
differ is that the former attend a charter school, while the latter consist of students attending a 
traditional public school. 
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Appendix Figure 1: Comparison of Starting Reading Scores of Matched Charter Students and VCRs 


Two-sample t test v^ith unequal variances 


Group 1 

Obs 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval] 

Matched | 
VCR 1 

122,702 

122,702 

-.4353565 

-.4356375 

.0025198 

.0025163 

.8826718 

.8814316 

-.4402954 

-.4405694 

-.4304177 

-.4307056 

combined | 

245,404 

-.435497 

.0017805 

.8820501 

-.4389868 

-.4320072 

diff 1 


.0002809 

.0035611 


-.0066987 

.0072606 

diff = 
Ho: diff = 

mean(Matched) - mean 
0 

(VCR) 

Welch's degrees 

t = 

of freedom = 

0.0789 

245404 

Ha: diff < 0 

Pr(T < t) = 0.5314 

Ha: diff ! 
Pr(|Tl > |t|) = 

= 0 

0.9371 

Ha: diff > 0 
Pr(T > t) = 0.4686 


Appendix Figure 2: Comparison of Starting Math Scores of Matched Charter Students and VCRs 


Two-sample t test with unequal variances 


Group 1 

Obs 

Mean 

Std. Err. 

Std. Dev. 

[95% Conf. 

Interval] 

Matched | 

123,735 

-.508633 

.0023221 

.8168229 

-.5131842 

-.5040817 

VCR 1 

123,735 

-.5089222 

.0023186 

.8156003 

-.5134667 

-.5043777 

combined | 

247,470 

-.5087776 

.0016407 

.8162102 

-.5119934 

-.5055618 

diff 1 


.0002893 

.0032815 


-.0061424 

.0067209 

diff = 

mean(Matched) - mean(VCR) 


t 

= 0.0881 

Ho: diff = 

0 


Welch's degrees of freedom 

= 247469 

Ha: diff < 0 


Ha: diff != 

0 

Ha: diff > 0 


Pr(T < t) = 0.5351 Pr(|T| > |t|) = 0.9298 Pr(T > t) = 0.4649 
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Measuring Academic Growth 

With four years of data, each subject-grade-year group of scores has slightly different mid-point 
averages and distributions. For end-of-course assessments (EOCs) there are only subject-year groups 
because EOCs are not grade specific. This means a student takes this assessment after completing the 
course, no matter what grade she is in. In our study, scores for all these separate tests are transformed 
to a common scale. All test scores have been converted to standardized scores to fit a "bell curve", in 
order to allow for year-to-year computations of growth.^'* 

When scores are standardized, every student is placed relative to their peers in the entire state of 
Pennsylvania. A student scoring in the 50th percentile in Pennsylvania receives a standardized score of 
zero, while a standardized score of one would place a student in the 84th percentile. Students who 
maintain their relative place from year to year would have a growth score of zero, while students who 
make larger gains relative to their peers will have positive growth scores. Conversely, students who 
make smaller academic gains than their peers will have negative growth scores in that year. 

Models for Analysis of the Charter School Impact 

After constructing a VCR for each charter student, we then set out to develop a model capable of 
providing a fair measure of charter impact. The National Charter School Research Project provided a 
very useful guide to begin the process^^ First, it was useful to consider student growth rather than 
achievement. A growth measure provided a strong method to control for each student’s educational 
history as well as the many observable differences between students that affect their academic 
achievement. The baseline model included controls for each student’s grade, race, gender, poverty 
status, special education status, English Language Learner status, and whether he was held back the 
previous year. The literature on measuring educational interventions^ found that the best estimation 
techniques must also include controls for baseline test scores. Each student’s prior year test score is 
controlled for in our baseline model. Additional controls are also included for year and period (1"‘ year 
in charter, 2"'' year in charter, etc.). The study’s baseline model is presented below. 

+ P^i,t + P^t + y^i.t + ^i,t 
where the dependent variable is 

^i.t = ^i.t ~ ^i.t-1 ( 2 ) 


For each subject-grade-year set of scores, scores are centered around a standardized midpoint of zero, which 
corresponds to the actual average score of the test before transformation. Then each score of the original test is 
recast as a measure of variation around that new score of zero, so that scores that fall below the original 
average score are expressed as negative numbers and those that are larger receive positive values. 

Betts, J. and Hill, P. et al. (2006). “Key Issues in Studying Charter Schools and Achievement: A Review and 
Suggestions for National Guidelines.” National Charter School Research Project White Paper Series, No. 2. 

Betts, J. and Tang, Y. (2011) “The Effect of Charter Schools on Student Achievement: A Meta-Analysis of the 
Literature." National Charter School Research Project. 
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And/Ait is the state-by-test z-score for student i in period t; Ain is the state-by-testz-score for student i in 
period t- l;/i,t is a setof control variables for student characteristics and period, /t is a year fixed effect, 
Cisa vector of variables for whether student i attended a charterschool and what type of charter school 
in period t, and £ is the error term. Errors are clustered around charters schools and their feeder 
patterns as well. 

In addition to the baseline model above, we explored additional interactions beyond a simple binary 
to indicate charter enrollment. These included both “double” and “triple” interactions between the 
charter variable and student characteristics. For example, to identify the impact of charter schools on 
different racial groups, we estimate models that break the charter variable into “charter_black,” 
“charter_hispanic,” etc. To further break down the impact of charters by race and poverty, the variables 
above were split again. For example, black students in charter schools are split further into students 
who live in poverty (“charter_black_poverty”) and those that do not (“charter_black_nonpoverty”). 

Presentation of Results 

In this report, we present the impacts of attending charter schools in terms of standard deviations. The 
base measures for these outcomes are referred to in statistics as z-scores. A z-score of 0 indicates the 
student’s achievement is average for his or her grade. Positive values represent higher performance 
while negative values represent lower performance. Likewise, a positive effect size value means a 
student or group of students has improved relative to the students in the state taking the same exam. 
This remains true regardless of the absolute level of achievement for those students. As with the z- 
scores, a negative effect size means the students have on average lost ground compared to their peers. 

It is important to remember that a school can have a positive effect size for its students (students are 
improving) but still have below-average achievement. Students with consistently positive effect sizes 
will eventually close the achievement gap if given enough time; however, such growth might take 
longer to close a particular gap than students spend in school. 

While it is fair to compare two effect sizes relationally (i.e., 0.08 is twice 0.04), this must be done with 
care as to the size of the lower value. It would be misleading to state one group grew twice as much as 
another if the values were extremely small such as 0.0001 and 0.0002. 

Finally, it is important to consider if an effect size is significant or not. In statistical models, values which 
are not statistically significant should be considered as no different from zero. Two effect sizes, one 
equal to .001 and the other equal to .01, would both be treated as no effect if neither were statistically 
significant. 

To assist the reader in interpreting the meaning of effect sizes, we include an estimate of the average 
number of days of learning required to achieve a particular effect size. This estimate was calculated by 
Dr. Eric Hanushek and Dr. Margaret Raymond based on the latest (2017) and 8**’ grade test scores 
from the National Assessment of Educational Progress (NAEP). Using a standard 180-day school year, 
each one standard deviation (s.d.) change in effect size was equivalent to 590 days of learning in this 
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study. The values in Tables are updated from past reports using more recent NAEP scores, which show 
slower absolute annual academic progress than earlier administrations.^^ 

In order to understand “days of learning,” consider a student whose academic achievement is at the 
50th percentile in one grade and also at the 50th percentile in the following grade the next year. The 
progress from one year to the next equals the average learning gains for a student between the two 
grades. That growth is fixed as 180 days of effective learning based on the typical 180-day school year. 

We then translate the standard deviations of growth from our models based on that 180-day average 
year of learning, so that students with positive effect sizes have additional growth beyond the expected 
180 days of annual academic progress while those with negative effect sizes have fewer days of 
academic progress in that same 180-day period. 


Hanushek, Eric A. P.E. Peterson, & L. Woessmann (2012). Achievement Growth: International and U.S. State 
Trends in Student Performance. Education Next, Vol. 12,1-35. 
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